Facial makeup has a long history. There are many techniques, categories and products related to makeup or cosmetics. Cosmetics are used to hide facial flaws and appear more attractive. With these advancements, the use of makeup is socially fashionable in every aspect of our lives. On the other hand, the use of makeup poses a significant challenge to biometric systems. The face recognition problem has attracted a tremendous amount of research over the past decade [39] and has been significantly improved. However, there are still several factors that challenge the performance of face recognition system at this stage, which include age [30], spoof [36] and facial makeup. Facial makeup is capable of altering and hiding one's original appearance, which makes some recognition or verification tasks more difficult. In a most recent paper, Dantcheva et al. [2] discussed the negative impact introduced by facial cosmetics to the face recognition problem.
Research on makeup recommendation systems has also increased recently. In the ACM Multimedia 2013 best paper [1], Liu, et al. developed a system for hairstyle and facial makeup recommendation and synthesis. Their work is based on a facial beauty evaluation algorithm. They applied candidate makeup onto an original face and recommended to users the candidate makeup that resulted in a highest beauty score. This system produces appealing results but still has a lot of limitation, such as it can only deal with a face without makeup.
Compared with work on makeup recommendations, research dealing with an already made up face image is even rarer. Dantcheva, et al. [2] is the first work that explicitly established the impact of facial makeup on a face recognition system. They assembled two datasets, YouTube MakeUp (YMU) database and Virtual MakeUp (VMU) database, then tested the recognition performance before and after makeup with three face recognition methods: Gabor wavelets, Local Binary Pattern and the commercial Verilook Face Tookit.
Based on this work, there are two papers that focus on a face with makeup. In [3], the presence of makeup in face images is detected based on a feature vector that contains shape, texture and color information. The other paper [4] deals with the verification problem. They extract features from both a face with makeup and a face without makeup, then do the face matching based on correlation mapping.
Facial beauty and its measurement have been widely debated for centuries. In the psychology community, many research efforts have attempted to find some biologically based standards common to humans from different cultures, genders and ages. Some good candidates for these kinds of standards include the idea of golden ratio [17], facial thirds or facial trisection Pi, averageness [12], and symmetry [15]. More recently, research in this area has shifted to computer science, because of the need for more complex feature representations. More detailed research survey in human science is provided by Rhodes [21].
It is still in the early stages for using machines to predict attractiveness, and only a few works have been published, most of which by now are “geodesic ratio” based methods. Ever since the preliminary feature-based facial beauty scoring system proposed by Aarabi, et al. [11], various geometrical features are extracted to determine attractiveness based on facial symmetry, golden ratios, or neoclassical canons. Although these methods produce promising results, they all suffer from: (1) heavy duty use of landmarks annotation, and (2) non-unified criteria for attractiveness. Therefore, a fully automatic paradigm learned by machine has not been achieved.
The first attempt to do appearance-based attractiveness prediction is from Whitehill, et al. [24]. They used eigenface and Gabor filter analysis on more than 2000 photographs using ε-SVM (support vector machine). Sutic et al. [22] used eigenfaces with different classification methods such as KNN (k-nearest neighbors) and AdaBoost (adaptive boosting). Gray et al. [16] built a multiscale model to extract features to feed into a classical linear regression model for predicting facial beauty. In the recent work of Haibin [25], a cost-sensitive ordinal regression is proposed to categorize face in beauty order.
Research into facial beauty has recently drawn attention in research with pattern recognition and computer vision techniques. However, research is mainly focused on face beauty estimation, while the research related to facial makeup is still quite limited.
In the machine learning field more generally, recent research has led to the rapid growth in the theory and application of dictionary learning [42] and low-rank representation [33]. The performance of problems such as image classification has improved with a well-adapted discriminative low-rank dictionary [35, 32]. In the cross-modal dictionary learning literature, Wang et al. [41] proposed semi-coupled dictionary learning to do image super-resolution. This work has not, however, been applied to the makeup detection problem or to perform makeup reversion or removal.